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ABSTRACT 


Methods  to  individually  allocate  credit  to  multiple 
sensors  assist  Army  decision  makers  during  Objective  Force 
development  and  integration.  Sensor  credit  apportionment 
supports  a  knowledge-based  common  operating  picture  by 
providing  a  means  to  assess  and  compare  the  variety  of 
automated  and  human  sources  that  contribute  to  the  common 
operating  picture  (U.S.  Army  White  Paper,  2001)  .  This 
thesis  develops  and  assesses  such  a  method.  The  method 
determines  the  contribution  of  individual  sensors  (persons 
or  platforms  that  provide  specific  target  information)  to 
successful  commander  decision  making.  The  method  utilizes 
data  fusion,  decision  analysis,  and  information  quality  to 
credit  each  sensor  according  to  the  benefit  they  provide  a 
commander.  A  stochastic  simulation  provides  the  means  to 
generate  an  environment  in  which  to  test  and  compare  sensor 
performance,  and  to  assess  the  credit  apportionment  method. 
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EXECUTIVE  SUMMARY 


This  thesis  develops  and  assesses  a  method  and  working 
model  to  determine  the  contribution  of  individual  sensors 
(persons  or  platforms  that  provide  specific  target 
information)  to  critical  information  gathering  and  to 
commander  decision  making.  This  thesis  addresses  credit 
apportionment  (dividing  and  assigning)  to  specific  sensors 
for  satisfying  an  information  requirement  (Bauman,  2001). 

A  United  States  Army  white  paper,  titled  "Concepts  for 
the  Objective  Force,"  states  that  the  United  States  Army 
established  Transformation  as  its  focus.  Transformation  is 
a  process  changing  the  Army  "...into  a  force  that  is 
strategically  responsive  and  dominant  at  every  point  on  the 
spectrum  of  conflict"  (U.S.  Army  White  Paper,  2001)  .  The 
Army  calls  this  future  force  the  Objective  Force. 

The  Objective  Force  anticipates  near-perfect 
situational  awareness:  "...Objective  Force  Units  will  see 
first..."  (U.S.  Army  White  Paper,  2001)  .  Seeing  first 
includes  "...detecting,  identifying,  and  tracking  the 
individual  components  of  enemy  units"  (U.S.  Army  White 
Paper,  2001).  Near-perfect  situational  awareness  helps 
commanders  make  critical  decisions  using  information  based 
more  on  facts  than  assumptions,  and  also  reduces  the  risk 
involved  in  decision  making.  Near-perfect  situational 
awareness  requires  accurate,  robust  information  sources  (or 
sensors)  .  It  also  requires  a  "...synthesized  common  picture 
of  the  battlefield,  the  common  operational  picture"  (U.S. 
Army  White  Paper,  2001)  .  Sensors,  and  the  common 


xvii 


operational  picture  they  create 
components  of  the  Objective  Force. 


make  up  critical 


Therefore,  the  Army  needs  methods  to  assess  and 
compare  sensors  in  terms  of  their  contribution  to  the 
common  operational  picture.  This  thesis  provides  one 
method  to  accomplish  such  an  evaluation. 

The  credit  apportionment  phenomenon  of  fusing 
information  from  separate,  distinct  sensors  to  create  a 
common  operational  picture  in  order  to  inform  a  decision 
maker,  and  then  apportion  credit  to  the  sensors  according 
to  their  individual  contributions  to  the  decision,  is  an 
emerging,  complex  area  of  interest  depicted  in  the  figure 
below . 


Figure  1.  Multiple  Sensor  Credit  Apportionment  Phenomenon. 

(After  Bauman,  2001) 

Examination  of  the  credit  apportionment  phenomenon 
first  requires  knowledge  of  its  process.  Then  it  demands 
developing  a  working  model,  or  algorithm.  Finally,  it 
requires  consideration  of  data  requirements  for  the  model. 

A  clearly  defined  understanding  of  the  interactions  at 
work  in  the  credit  apportionment  phenomenon  does  not  yet 
exist.  As  a  result,  a  significant  portion  of  this  research 
effort  attempts  to  communicate  the  essence  of  the 
apportioning  credit  phenomenon  and  its  link  to  data  fusion 
and  information  quality. 
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The  assessment  of  the  multiple  sensor  credit 
apportionment  method  includes  conducting  five  experiments 
with  various  settings  of  three  factors.  The  factors  are 
the  presence  or  absence  of  sensors,  and  the  settings 
represent  the  sensor  characteristics.  Each  experiment 
provides  verification  of  the  underlying  calculations,  as 
well  as  example  results  based  on  estimated  input 
parameters.  The  primary  input  parameters  include  sensor 
characteristics  and  target  attribute  presence  data. 

Methods  to  individually  allocate  credit  to  multiple 
sensors,  such  as  the  method  in  this  thesis,  may  assist  Army 
decision  makers  during  Objective  Force  development  and 
integration.  Being  able  to  credit  sources  (or  sensors)  for 
the  benefit  they  provide  allows  decision  makers  a  means  to 
compare  sensors  during  development.  In  terms  of  force 
integration,  methods  to  credit  sensors  also  allow  decision 
makers  ways  to  find  the  best  mix  of  sensors  for  a 
particular  force. 
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INTRODUCTION 


I . 


A.  THESIS  PURPOSE 

This  thesis  develops  and  assesses  a  method  and  working 
model  to  determine  the  contribution  of  individual  sensors 
(persons  or  platforms  that  provide  specific  target 
information)  to  critical  information  gathering  and  to 
commander  decision  making.  This  thesis  addresses  credit 
apportionment  (dividing  and  assigning)  to  specific  sensors 
for  satisfying  an  information  requirement  (Bauman,  2001). 

B.  SIGNIFICANCE 

A  United  States  Army  white  paper,  titled  "Concepts  for 
the  Objective  Force,"  states  that  in  the  document.  The  Army 
Vision,  the  United  States  Army  established  Transformation 

as  its  focus.  The  Army  Transformation  is  a  process 
changing  the  Army  "...into  a  force  that  is  strategically 
responsive  and  dominant  at  every  point  on  the  spectrum  of 
conflict"  (U.S.  Army  White  Paper,  2001)  .  The  Army  calls 
this  future  force  the  Objective  Force. 

Methods  to  individually  allocate  credit  to  multiple 
sensors  may  assist  Army  decision  makers  during  Objective 

Force  development  and  integration.  Seven  force 

characteristics  guide  Objective  Force  development  (U.S. 
Army  White  Paper,  2001)  : 

•  Responsive 

•  Deployable 

•  Agile 

•  Versatile 


Lethal 


1 


Survivable 


• 

•  Sustainable 

The  Agile  characterist ic  represents  possibly  the  most 
direct  beneficiary  of  sensor  credit  apportionment  methods. 
The  white  paper  defines  Objective  Force  Agility  as  the 
ability  to  quickly  transition  among  various  types  of 
operations,  such  as  from  support  operations  to  warfighting 
and  back  again  (U.S.  Army  White  Paper,  2001)  .  The  paper 
states  information  superiority  through  a  knowledge-based 
common  operational  picture  constitutes  the  essential 
element  to  enabling  Agility  in  the  Objective  Force  (U.S. 
Army  White  Paper,  2001)  .  A  "common  operational  picture"  is 
a  "...synthesized,  common  picture  of  the  battlefield..."  (U.S. 
Army  White  Paper,  2001)  .  "Knowledge-based"  means  a  common 
operational  picture  consists  of  "...near-real  time..." 
information  "...from  a  variety  of  automated  and  human 
sources..."  (U.S.  Army  White  Paper,  2001).  Information 
superiority  through  a  common  operational  picture  enables 
Agility,  because  it  expedites  the  decision-action  cycle. 

Sensor  credit  apportionment  supports  a  knowledge-based 
common  operating  picture  by  providing  a  means  to  compare 
and  assess  the  "...variety  of  automated  and  human  sources..." 
that  contribute  to  the  common  operating  picture  (U.S.  Army 
White  Paper,  2001) . 

Being  able  to  credit  sources  (or  sensors)  for  the 
benefit  they  provide  allows  decision  makers  a  means  to 
compare  sensors  during  development.  In  terms  of  force 
integration,  methods  to  credit  sensors  also  allow  decision 
makers  ways  to  find  the  best  mix  of  sensors  for  a 
particular  force. 
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C .  BACKGROUND 

The  United  States  Army  Military  Decision  making 

Process  consists  of  the  following  steps  (Field  Manual  101- 
5,  1997) : 

•  Receipt  of  Mission 

•  Mission  Analysis 

•  Course  of  Action  Development 

•  Course  of  Action  Analysis 

•  Course  of  Action  Comparison 

•  Course  of  Action  Approval 

•  Orders  Production 

•  Rehearsal 

•  Execution  and  Assessment 

This  thesis  focuses  on  Execution  and  Assessment. 
During  Execution,  a  commander  uses  reports  of  Priority 
Information  Requirements  (PIR) —"information  about  the 
enemy"— in  order  to  make  execution  decisions  (Eield  Manual 
101-5,  1997)  .  The  commander  must  make  decisions  with  an 

incomplete  view  of  the  battlefield.  The  commander  must 
choose  PIRs  carefully  in  order  to  gain  critical  information 
needed  to  make  the  best  timely  execution  decision. 

The  Army  acknowledges  commanders  must  make  execution 
decisions  without  a  complete  picture  of  battlefield 
reality.  In  Eield  Manual  101-5,  the  Army  emphasizes 
commanders  must  "make  critical  decisions  using  information 
based  more  on  assumptions  than  facts,"  and  they  must  "be 
resolute  in  accepting  risk  and  be  willing  to  make  decisions 
based  only  on  information  immediately  available"  (Eield 
Manual  101-5,  1997). 
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However,  the  future  Army  Objective  Force  anticipates 
near-perfect  situational  awareness:  "At  the  tactical 

level.  Objective  Force  Units  will  see  first,  understand 
first,  act  first  and  finish  decisively  as  the  means  to 
tactical  success"  (U.S.  Army  White  Paper,  2001).  Seeing 
first  includes  "...detecting,  identifying,  and  tracking  the 
individual  components  of  enemy  units"  (U.S.  Army  White 
Paper,  2001).  Near-perfect  situational  awareness  helps 
commanders  make  critical  decisions  using  information  based 
more  on  facts  than  assumptions,  and  also  reduces  the  risk 
involved  in  decision  making.  Near-perfect  situational 
awareness  requires  accurate,  robust  information  sources  (or 
sensors)  .  It  also  requires  a  "...synthesized  common  picture 
of  the  battlefield,  the  common  operational  picture." 
Sensors  and  the  common  operational  picture  they  create  make 
up  critical  components  of  the  Objective  Force  (U.S.  Army 
White  Paper,  2001)  . 

Therefore,  the  Army  needs  methods  to  assess  and 
compare  sensors  in  terms  of  their  contribution  to  the 
common  operational  picture.  This  thesis  provides  one 
method  to  accomplish  such  an  evaluation  of  the  sensors. 

D.  KNOWLEDGE,  ALGORITHMS,  AND  DATA 

A  framework  to  examine  models  and  simulations 
implemented  by  the  Training  and  Doctrine  Command  Analysis 
Center-Monterey  (TRAC-Monterey)  and  the  United  States  Army 
Materiel  Systems  Analysis  Activity  (AMSAA)  examines 
"knowledge"  of  the  modeled  phenomenon,  "algorithms"  used, 
and  "data"  required.  The  question  they  ask  is:  what 

knowledge,  algorithms,  and  data  are  needed  to  adequately 
represent  the  phenomenon? 
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The  credit  apportionment  phenomenon  of  fusing 
information  from  separate,  distinct  sensors  to  create  a 
clearer  information  picture  in  order  to  inform  a  decision 
maker,  and  then  apportion  credit  to  the  sensors  according 
to  their  individual  contributions  to  the  decision,  is  an 
emerging,  complex  area  of  interest  depicted  in  the  figure 
below . 


Figure  2.  Multiple  Sensor  Credit  Apportionment  Phenomenon. 

(After  Bauman,  2001) 

Examination  of  the  credit  apportionment  phenomenon 
first  requires  knowledge  of  its  process.  Then  it  demands 
developing  a  working  model,  or  algorithm.  Finally,  it 
requires  consideration  of  data  requirements  for  the  model. 

1 .  Knowledge 

Very  little  published  information  exists  about 
multiple  sensor  credit  apportionment,  and  a  clearly  defined 
understanding  of  the  processes  at  work  in  the  credit 
apportionment  phenomenon  does  not  currently  exist.  As  a 
result,  a  significant  portion  of  this  research  effort 
attempts  to  communicate  the  essence  of  the  apportioning 
credit  phenomenon  and  its  link  to  data  fusion  and 
information  quality  (both  discussed  later  in  this  section) . 

2 .  Algorithm 

This  thesis  focuses  on  the  development  of  an 

appropriate  algorithm  that  fuses  detection  information  from 
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independent  sources  while  considering  information  quality 
in  order  to  make  a  decision,  assess  the  decision,  and 
apportion  credit.  This  complex  and  detailed  algorithm 
forms  the  basis  for  a  computer  simulation. 

Three  model  development  steps  make  up  the  algorithm 
development  process  for  this  thesis: 

•  determine  sensor  detection  capabilities, 

•  determine  information  quality  each  sensor 

provides , 

•  fuse  detection  capabilities  and  information  value 
from  multiple  sensors. 

a.  Determine  Sensor  Detection  Capabilities 

The  first  model  development  step  provides  the 
foundation  of  the  sensor  credit  method.  The  contribution 
of  individual  sensors  ultimately  traces  back  to  their  basic 
target  detection  capabilities.  In  order  to  determine 
sensor  target  detection  capabilities,  the  sensor  requires 
the  presence  probability  (probability  a  target  is  present) 
of  some  type  of  target.  In  addition,  sensor  target 

detection  capabilities  require  sensor  detection 
capabilities  of  target  attributes  (a  sensor  only  detects  a 
target  by  detecting  its  attributes.)  A  target  inherently 
possesses  some  measurable  attributes  (size,  shape, 
location,  activity,  etc.)  that  define  it  as  a  target.  This 
study  investigates  only  single  targets,  and  single 
detections  by  each  sensor.  Determining  sensor  detection 
capabilities  addresses  the  process  of  combining  the 
presence  of  a  target,  the  presence  of  target  attributes, 
and  the  sensor's  ability  to  detect  those  attributes. 
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b.  Determine  Each  Sensor's  Information  Quality 


The  second  model  development  step  determines  how 
much  value  the  sensors'  detection  information  holds  for  a 
commander  by  calculating  an  information  quality  value  for 
each  sensor.  Knowing  a  sensor's  detection  capability  fails 
to  provide  enough  basis  to  determine  its  contribution  to 
satisfying  an  information  requirement.  Therefore,  it  is 
important  to  know  whether  the  detection  information  holds 
any  value  to  a  commander.  Dr.  Walter  Perry,  RAND 
Corporation,  states,  "Information  has  value  if  it  informs 
the  commander  and  thereby  adds  to  his  knowledge  of  the 
combat  situation"  (Perry,  2000).  He  also  states  that 
information  quality  consists  of  accuracy,  timeliness,  and 
completeness  (Perry,  2000).  His  component  breakdown  of 
information  quality  into  accuracy,  timeliness,  and 
completeness  forms  the  basis  of  the  second  step 
(determining  information  quality  each  sensor  provides)  of 
the  sensor  credit  apportionment  method. 

c.  Fuse  Information 

The  third  model  development  step  fuses  the 
results  from  step  one  (sensor  detection  capabilities)  and 
step  two  (sensor  information  quality)  for  multiple  sensors. 
Fusing  information  in  this  manner  from  multiple  sensors 
attempts  to  represent  a  commander' s  thought  process  of 
combining  all  available  information  to  create  a  clearer 
battlefield  common  operational  picture  when  deciding 
whether  or  not  to  execute  a  course  of  action.  After  fusing 
the  information,  a  commander  makes  a  decision  based  on  the 
fused,  sensor-provided  information.  The  information  yields 
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either  a  correct  or  an  incorrect  decision.  A  sensor  that 
provides  information  leading  to  a  correct  decision  receives 
credit  for  its  contribution.  Conversely,  a  sensor  that 
provides  information  leading  to  an  incorrect  decision 
receives  no  credit.  (For  this  thesis,  each  sensor  provides 
either  good  information  or  poor  information.  It  cannot 
renege  providing  information.)  The  correctness  of  the 
commander' s  decision  based  on  the  actual  target  state  and 
sensor-provided  information  provides  the  measure  for 
comparing  performance  of  various  sensor  combinations. 

3 .  Data 

Consideration  of  data  requirements  discovers  that  the 
credit  apportionment  model  in  this  thesis  depends  heavily 
on  input  data  (59  parameters  when  experimenting  with  three 
sensors) .  The  majority  (48  parameters)  of  the  input 
parameters  includes  sensor  characteristics  and  target 
attribute  presence  data.  The  other  eleven  parameters 
include  parameters  for  generating  a  target  presence 
probability  from  a  uniform  distribution,  and  parameters  for 
generating  initial  information  quality  values  from  normal 
and  exponential  distributions. 

For  this  thesis,  it  is  sufficient  that  the  input  data 
are  estimated.  Therefore,  while  this  model  does  not 
provide  realistic  performance  of  each  sensor,  it  does 
provide  insight  about  the  relationships  between  sensors,  an 
operational  decision,  and  appropriate  credit  given  to  the 
sensors.  Also,  while  this  model  uses  estimated  input  data, 
further  research  and  field  experimentation  may  yield 
realistic  input  data.  For  this  thesis,  we  estimated  all 
the  input  parameters.  Regardless  of  the  "goodness"  of  the 


input  data,  the  multiple  sensor  credit  apportionment  method 
in  this  thesis  provides  insight  about  apportioning  credit 
to  sensors  based  on  their  performance. 

E .  SCOPE 

This  thesis  focuses  on  the  development  and  assessment 
of  a  sensor  crediting  simulation  model.  The  model  gives 
credit  to  each  sensor  for  its  role  in  assisting  a  commander 
make  correct  mission  execution  decisions. 

This  study  uses  independent  sensors  in  order  to 
simplify  the  credit  apportionment.  Independent  sensors 
report  information  based  solely  on  their  own  detections  and 
inherent  abilities.  Commanders  then  fuse  the  sensors' 
independent  information  into  a  common  operational  picture. 

Although  this  thesis  develops  a  method  to  apportion 
credit  to  sensors,  it  does  not  develop  an  optimal  sensor. 
Additionally,  this  thesis  does  not  determine  an  optimal 
data  fusion  method.  Finally,  this  thesis  limits  the 
definition  of  information  quality  to  three  components: 
accuracy,  timeliness,  and  completeness.  The  next  chapter 
provides  definitions  of  the  information  components  used  in 
this  thesis. 

F .  GENERAL  METHODOLOGY 


The  figure  above  illustrates  a  general  methodology  for 
representing  the  multiple  sensor  credit  apportionment 
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phenomenon.  The  Activity  node  represents  an  activity 
occurring  in  an  area  of  interest.  Various  sensors  detect 
or  do  not  detect  the  activity.  The  sensors'  reports  are 
fused  (Fuse  Info  node)  in  order  to  support  a  decision 
(Decision/Assess  node)  to  execute  a  mission  or  to  continue 
monitoring  activity  in  the  area  of  interest.  Assessment  of 
the  execution  decision  (Decision/Assess  node)  reveals  how 
valuable  the  sensors'  information  truly  was.  Each  sensor's 
contribution  to  correct  mission  execution  provides  the 
basis  for  each  sensor  to  receive  appropriate  credit  in  the 
Apportion  Credit  node.  The  arcs  between  the  Apportion 
Credit  node  and  the  Fuse  Info  node  represent  learning  (or 
experience)  of  the  commander  after  each  assessment  and 
ranking  of  sensors.  The  commander's  next  decision  will  be 
flavored  by  his  previous  experience  with  the  sensors. 
Previously  successful  sensors  bias  the  commander  toward 
favoring  their  results.  Similarly,  previously  unsuccessful 
sensors  bias  the  commander  away  from  favoring  their 
results.  For  this  thesis,  however,  we  assume  an  unbiased 
commander  for  all  decisions  (no  learning  or  experience)  . 
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II.  MODEL  DEVELOPMENT 


A.  OVERVIEW 

Combat  models  and  simulations  traditionally  develop 
algorithms  and  code  that  acceptably  represent  physics-based 
phenomena.  Specifically  these  algorithms  calculate,  or 
determine,  line  of  sight,  detection  probability,  successful 
engagement  probability,  or  attrition  rates.  These 
phenomena,  critical  to  the  conduct  of  force-on-force 
modeling,  are  well  documented  in  numerous  accepted  models. 

However,  recent  interest  in  combat  modeling 
development  mirrors  the  Army  Transformation  addressed  in 
the  previous  chapter.  One  of  the  areas  of  interest  is  data 
fusion— an  extremely  complex  phenomenon.  A  human  being 
stands  as  the  best  example  of  effective  data  fusion: 

Humans  naturally  apply  this  ability  to  combine 
data  (sight,  sound,  scent,  touch)  from  the  body's 
sensors  (eyes,  ears,  nose,  fingers)  with  prior 
knowledge  to  assess  the  world  and  the  events 
occurring  about  them.  Because  the  human  senses 
measure  different  physical  phenomenon  over 
different  spatial  volumes  with  different 
measurement  characteristics,  this  process  is  both 
complex  and  adaptive.  The  conversion  of  the 
data... into  a  meaningful  perception  of  the 
environment  requires  a  large  number  of  distinct 
intelligence  processes  and  a  base  of  knowledge 
sufficient  to  interpret  the  meaning  of  the 
properly  combined  data  (Waltz  and  Llinas,  1990)  . 

As  mentioned  earlier,  the  Army' s  desire  to  develop  a 
battlefield  common  operational  picture  leads  to  the  current 
interest  in  data  fusion  and  credit  apportionment.  Data 
fusion  represents  the  process  required  to  achieve  a  common 
operational  picture.  Data  fusion  combines  "...elements  of 
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raw  data  from  different  sources  into  a  single  set  of 
meaningful  information  that  is  of  greater  benefit  than  the 
sum  of  its  contributing  parts"  (Waltz  and  Llinas,  1990)  . 
The  Objective  Force  common  operational  picture  is  the 
"...single  set  of  meaningful  information..."  desired  by 
military  commanders  to  expedite  the  decision-action  cycle 
(Waltz  and  Llinas,  1990)  . 

Fusion  of  information,  or  data,  from  multiple  sensors 
directly  leads  to  a  common  operational  picture,  while 
multiple  sensor  credit  apportionment  provides  a  method  to 
know  which  sensors  provide  beneficial  input  to  the  common 
operational  picture. 

B .  GENERAL  MODEL 


- Target  presence 

Sensor  Deteetion  Capabilities 
Sensor  Information  Quality 

i 

Fuse  Information  ** - 


Component  I 


Component  II 


*  Deeisions 

1 


V  Component  III 


MOE  J 

Figure  4.  General  Credit  Apportionment  Model 

The  figure  above  summarizes  the  credit  apportionment 
model  for  this  thesis.  The  following  list  displays  the 
subcomponents  of  the  general  credit  apportionment  model 
shown  above . 

•  Target  Presence 

^  generate  target  presence  probability 
^  calculate  attribute  presence  probability 
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•  Sensor  Detection  Capabilities 

^  calculate  probability  sensor  detects  attributes 
^  calculate  probability  sensor  detects  target 

•  Sensor  Information  Quality 

^  define  sensor  detection  outcome 

^  define  expected  return  value 

^  define  information  quality  components 

^  generate  sensor  information  accuracy 

^  generate  sensor  information  timeliness 

^  calculate  sensor  information  completeness 

^  calculate  total  information  quality  for  each 
sensor 

•  Fuse  Information 

^  define  and  illustrate  decision  tree 

•  Decisions 

>  generate  actual  target  state 

^  generate  sensor  "decision"  of  the  target  state 
^  calculate  commander' s  decision 

•  Measure  of  Effectiveness  (MOE) 

^  count  number  of  times  actual  target  state, 
sensor  "decision",  and  commander's  decision  all 
agree 

The  figure  below  provides  a  more  detailed  look  at  the 
credit  apportionment  model. 
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Figure  6.  Credit  Apportionment  Model  Components 

Component  I  consists  of  sensor  and  target  input. 
Component  I  includes  Target  Presence  and  Sensor  Detection 
Capabilities  from  the  general  model  presented  above. 
Similarly  Component  II  consists  of  information  quality 
input,  and  it  includes  Sensor  Information  Quality  from  the 
general  model.  Component  III  consists  of  decisions  and 
output,  and  it  includes  Fuse  Information,  Decisions,  and 
MOE  from  the  general  model. 
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c. 


MODEL  COMPONENT  I,  SENSOR  INPUT 
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Model  Component  I,  Sensor  Input 


1 .  Target  Presence 

The  Law  of  Total  Probability, 

Pr(5)  =  Pr(5  1 * Pr(,4j) H - 1- Pr(5  |  ,4^) * Pr(,4^)  ,  establishes  the  basis 

of  Model  Component  I  (Devore,  2000)  .  The  outputs  of  Model 
Component  I,  sensor  detection  probabilities,  make  up  part 
of  the  input  for  Components  II  and  III. 

a.  Target  Presence  Probability 

To  obtain  sensor  detection  probabilities,  we 
begin  with  the  probability  a  target  (threat  entity)  is 
present,  Pr(  target  present)  .  Its  value  comes  from  a 

uniform(a  =  0.0, b  =  0.75)  draw.  We  keep  those  parameters  constant 
for  this  thesis,  but  any  combination  of  parameters  between 
a=0 . 0  and  b=l .  0  are  appropriate  for  this  model.  For  each 
replication  a  random  draw  generates  the  probability  a 
target  is  present  (available  for  detection) .  The 

probability  a  target  is  present  also  becomes  input  for 
Model  Component  III  for  determining  the  actual  target 
state . 

ib.  Attribute  Presence  Probability 

Next  we  calculate  the  probability  a  target 
attribute  is  present,  Pr(  attribute  present) ,  using  the  law  of  total 
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probability.  As  stated  previously,  a  target  inherently 
possesses  some  measurable  attributes  (size,  shape, 
location,  activity,  etc.)  that  define  it  as  a  target. 
Therefore,  the  probability  an  attribute  is  present  is  a 
function  of  whether  the  target  is  present  or  not.  However, 
presence  of  a  target  attribute  does  not  guarantee  target 
presence . 

Pr( attribute  present)  =  Pr( attribute  present  |  target  present)  * 

Pr( target  present)  + 

Pr( attribute  present  |  target  NOT  present)  * 

Pr( target  NOT  present) 

For  this  model,  the  two  conditional  probabilities 
in  the  previous  equation  are  prior  probabilities  obtained 
from  a  draw  from  a  triangle  distribution.  We  estimated  the 
triangle  distribution' s  parameters  (Appendix  B)  . 
Experimentation  or  historical  data  should  provide  more 
realistic  values;  however,  the  triangle  distribution 
provides  reasonable  values  in  the  "...absence  of  data..."  (Law 
and  Kelton,  2000)  . 

We  calculate  Pr(attribute  NOT  present)  similarly: 

Pr( attribute  NOT  present)  =  Pr( attribute  NOT  present  |  target  present)  * 

Pr( target  present)  + 

Pr( attribute  NOT  present  |  target  NOT  present)  * 
Pr( target  NOT  present) 

2 .  Sensor  Detection  Capabilities 

a.  Probability  Sensor  Detects  Attributes 

Next  we  use  Pr( attribute  present)  and  Pr( attribute  NOT  present) 

to  calculate  the  probability  a  sensor  detects  a  target 
attribute,  Pr(sensor  deteets attribute)  .  A  sensor  is  any  person  or 
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platform  that  provides  specific  target  information.  The 
probability  a  sensor  detects  an  attribute  is  a  function  of 
whether  the  attribute  is  present  or  not.  However, 

detection  of  an  attribute  does  not  guarantee  an  attribute 
is  actually  present.  As  before,  we  use  the  law  of  total 
probability : 

Pr(sensor  detects  attribute)  =  Pr(sensor  detects  attribute  |  attribute  present)  * 

Pr( attribute  present)  + 

Pr(sensor  detects  attribute  |  attribute  NOT  present)  * 

Pr( attribute  NOT  present) 

Sensor  specifications  obtained  from  experimental 
data  make  up  the  conditional  probabilities  in  the 
equations . 

We  calculate  Pr(sensor  NOT  detect  attribute)  similarly: 

Pr(sensor  NOT  detect  attribute)  =  Pr(sensor  NOT  detect  attribute  |  attribute  present)  * 

Pr( attribute  present)  + 

Pr(sensor  NOT  detect  attribute  |  attribute  NOT  present)  * 
Pr( attribute  NOT  present) 

The  probability  a  sensor  detects  a  target 
attribute  becomes  input  to  Model  Component  IT  for 
determining  initial  information  completeness  values  for 
each  sensor. 

ib.  Probability  Sensor  Detects  Target 

The  sensor  target  detection  probabilities 
represent  a  sensor' s  "decision"  about  whether  a  target 
actually  exists.  The  target  may  or  may  not  actually  be 
present,  so  a  sensor's  "decision"  may  or  may  not  be 
correct.  In  order  to  determine  the  sensor  target  detection 
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probabilities,  Pr(sensor  detects  target)  ,  we  use  Pr(sensor  detects  attribute) 
and  Pr(sensor  NOT  detect  attribute)  from  above.  Again  sensor 

specifications  obtained  from  experimental  data  make  up  the 
conditional  probabilities  below.  Using  the  law  of  total 
probability : 

Pr(sensor  detects  target)  =  Pr(sensor  detects  target  |  sensor  detects  attribute)  * 

Pr(sensor  detects  attribute)  + 

Pr(sensor  detects  target  |  sensor  NOT  detect  attribute)  * 
Pr(sensor  NOT  detect  attribute) 

The  sensor  detection  probabilities  become  input 
to  Model  Component  III  for  determining  each  sensor's 
"decision"  about  the  presence  of  a  target,  and  as  one  input 
for  the  decision  tree. 
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D.  MODEL  COMPONENT  II,  INFORMATION  QUALITY  INPUT 


I. 


Figure  8.  Model  Component  II,  Information  Quality  Input 

1 .  Sensor  Information  Quality 

Component  II  takes  sensors'  target  attributes 
detection  capabilities  and  randomly  generated  values  as 
inputs,  and  produces  expected  return  values  for  use  in  a 
decision  tree  in  Component  III.  The  expected  return  value 
for  each  sensor  detection  outcome  comes  from  a 
representation  of  information  quality  (defined  below) 
calculated  in  this  model  component.  We  now  define  the 
sensor  detection  outcomes  and  expected  return  values  in 
more  detail. 
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Sensor  Detection  Outcome 


a . 

First,  what  is  a  sensor  detection  outcome?  If  a 
specific  application  of  this  model  includes  two  sensors, 
then  an  example  sensor  detection  outcome  is  when  both 
sensors  detect  a  particular  target,  and  their  detection  is 
correct.  (For  this  research,  a  sensor  detection  outcome  is 
either  'detect'  or  'no  detect'.)  Another  example  is  when 
the  first  sensor  correctly  detects  a  particular  target,  but 
the  second  sensor  incorrectly  does  not  detect  it.  Two 
sensors  yield  four  different  sensor  detection  outcomes. 
The  number  of  mutually  exclusive  sensor  detection  outcomes 
is  determined  by  2^,  where  k  is  the  number  of  sensors. 

b.  Expected  Return  Value 

Second,  what  is  an  expected  return  value  for  a 
sensor  detection  outcome?  An  expected  return  value  in  this 
model  is  the  value  of  information  provided  by  a  sensor 
detection  outcome. 

Extending  the  example  of  two  sensors  from  above, 
if  both  sensors  correctly  detect  a  particular  target,  then 
that  sensor  detection  outcome  has  the  largest  expected 
return  value.  For  this  model,  if  both  sensors  correctly 
detect  a  particular  target,  then  the  expected  return  value 
has  a  large  magnitude  for  a  decision  to  attack  the  target. 
However,  the  same  detection  outcome  (both  sensors  correctly 
detect  the  same  target)  yields  a  zero  expected  return  value 
for  a  decision  to  not  attack  (because  if  both  sensors 
detect  the  same  target,  the  expected  return  for  not 
striking  should  be  very  low,  or  even  zero,  for  this  model) . 
One  additional  example:  if  one  sensor  incorrectly  detects 
a  target  (target  actually  not  present)  and  the  second 
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sensor  correctly  does  not  detect  a  target,  then  the 
expected  return  value  for  that  sensor  detection  outcome 
consists  of  only  the  information  value  for  the  second 
sensor  (the  sensor  which  reported  correctly) . 


Therefore, 

we  calculate 

expected  return 

values  as 

linear 

combinations 

of  the 

information  values 

of 

each 

sensor . 

As  highlighted  in 

the 

last  example  above. 

the 

linear 

combinations 

depend 

on 

the  correctness 

of 

each 

sensor' 

s  detection 

report . 

We 

refer  to  each 

sensor' s 

information  value  as  the  total  information  quality  for  that 
sensor.  All  the  following  portions  of  this  section  discuss 
the  calculations  needed  to  determine  each  sensor' s  total 
information  quality. 

c.  Information  Quality  Components 

In  this  thesis  we  assume  that  three  components 
determine  information  quality:  accuracy,  timeliness,  and 
completeness  (Perry,  2000).  This  thesis  defines 
information  quality  components  in  the  following  manner: 
Accuracy  encompasses  how  closely  a  sensor' s  report  of  an 
entity's  location  compares  to  the  actual  entity  location. 
Timeliness  indicates  whether  a  commander  receives 
information  early  enough  or  too  late  to  act.  Completeness 
indicates  the  degree  to  which  a  sensor  correctly  identifies 
an  entity's  attributes,  which,  in  turn,  leads  to  accurate 
entity  identification. 

Sensor  characteristics,  operator  and  decision¬ 
maker  training,  time  of  day,  and  any  of  many  other 
operational  conditions  influence  all  three  of  the 
information  quality  components.  Therefore,  we  attempt  to 


22 


group  and  estimate  those  operational  influences  as 
described  in  the  following  subsections. 

d.  Sensor  Information  Accuracy 

To  represent  a  sensor' s  accuracy  in  determining 
location,  we  first  generate  a  normal(// =  0,<T  =  1)  for  each  sensor 
to  represent  a  miss  distance  (radial  distance  that 
represents  the  sensor's  location  error) .  If  the  resulting 
random  miss  distance  is  small  (within  a  given  range) ,  then 
the  location  accuracy  value  is  high  (high  values  are  good) . 
Conversely,  if  the  miss  distance  is  large  then  the  location 
accuracy  value  is  low.  We  use  a  conditional  setting  to 
assign  values  to  sensors  with  miss  distances  within  set 
ranges.  The  ranges  correspond  to  standard  normal  random 
variables  (since  we  used  the  standard  normal  distribution) . 
The  best  miss  distances  occur  when  -0.4<z<0.4,  the  next 
best  occur  when  0.4  <  z  <  0.8  or  -0.8<z<-0.4,  and  so  on.  The 
worst  miss  distances  occur  for  z>1.6  or  z<-1.6.  For 
example,  if  a  sensor's  generated  miss  distance  is  z  =  0.35 , 
then  the  sensor  receives  an  accuracy  value  of  8.0.  And  if 
a  sensor's  generated  miss  distance  is  z  =  1.7,  then  the 
sensor  receives  an  accuracy  value  of  0.0.  We  set  the 
accuracy  values  in  order  to  distinguish  between  a  good  miss 
distance  (high  value)  and  a  poor  miss  distance  (low  value) . 

We  use  the  normal  distribution  for  the  miss 
distance  because  previous  applications  have  shown  it  is 
appropriate  for  representing  location  error  (Law  and 
Kelton,  2000) .  We  use  a  mean  of  zero  and  standard 
deviation  of  one  as  the  normal  distribution' s  parameters 
(Appendix  B)  to  illustrate  one  example  setting  of  those 
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parameters.  For  this  thesis,  we  held  those  constant,  but 
they  are  variable. 

e.  Sensor  Information  Timeliness 

To  represent  timeliness,  we  generate  an 
exponential(/l  =  0.2)  for  each  sensor  to  represent  the  amount  of 
time  (report  time)  from  sensor  detection  to  the  commander's 
receipt  of  the  information.  If  the  elapsed  time  is  small, 
then  the  timeliness  value  is  high  (high  is  good) ,  and 
conversely.  Like  sensor  accuracy  above,  we  use  a 

conditional  setting  to  assign  values  to  sensors  with  report 
times  within  set  ranges.  The  ranges  correspond  to 

reasonable  exponential  variables.  For  this  thesis  the  best 
report  times  occur  when  x<1.0,  the  next  best  occur  when 
1.0<x<5.0,  and  so  on.  The  worst  report  times  occur  for 
X  >  25.0  .  For  example,  if  a  sensor's  generated  report  time 
is  X  =  0.9 ,  then  the  sensor  receives  a  timeliness  value  of 
8.0.  And  if  a  sensor's  generated  report  time  is  x  =  27.3  , 
then  the  sensor  receives  a  timeliness  value  of  0.0.  We  set 
the  timeliness  values  in  order  to  distinguish  between  a 
good  (low  value)  and  a  poor  (high  value)  report  time. 

We  use  the  exponential  distribution  based  on  its 
applicability  to  represent  the  time  to  complete  a  task  (Law 
and  Kelton,  2000)  .  In  this  thesis,  we  use  a  constant  rate 
of  0.2  as  the  exponential  distribution's  parameter 
(Appendix  B)  to  illustrate  one  example  setting  of  that 
parameter.  Like  sensor  accuracy  discussed  above,  this 
parameter  is  also  variable. 

f.  Sensor  Information  Completeness 

To  represent  completeness  (degree  to  which  the 
sensor  correctly  detects  all  target  attributes) ,  we  input 


each  sensor' s  probability  of  detecting  each  target 
attribute  from  Component  I.  Next,  we  compare  a  sensor's 
attribute  detection  probability  to  a  draw  of  a 
uniform(a  =  0,b  =  1)  .  If  the  random  number  is  less  than  or  equal 
to  the  detection  probability,  then  that  sensor  detected  the 
attribute.  However,  that  does  not  mean  the  attribute  is 
actually  present.  Therefore,  we  compare  another 

uniform(a  =  0,b  =  1)  draw  to  the  probability  that  the  attribute 
is  present  in  order  to  determine  if  the  attribute  is 
present.  If  the  attribute  is  present  and  the  sensor 
detects  the  attribute,  the  sensor  receives  positive  value 
for  correctly  detecting  the  attribute.  The  sensor  also 
receives  positive  value  for  correctly  not  detecting  an 
attribute  when  an  attribute  is  not  present.  The  sensor 
receives  no  value  for  any  incorrect  attribute  detections. 
Therefore,  all  incorrect  attribute  detections  yield  no 
value  for  a  sensor,  while  each  correct  attribute  detection 
yields  positive  value  for  the  sensor.  For  example,  if  a 
sensor  correctly  detects  one  of  the  three  attributes  of  a 
target,  then  the  value  it  receives  (value  of  2.0)  is  less 
than  a  sensor  that  correctly  detects  all  three  target 
attributes  (value  of  8.0)  .  The  same  values  apply  to  the 
sensor  if  it  correctly  does  not  detect  an  attribute  when  an 
attribute  is  not  present.  In  other  words,  the  sensor 
receives  value  for  being  correct.  Since  this  model  uses 
three  attributes,  a  sensor  receives  different  levels  of 
value  based  on  how  many  attributes  the  sensor  correctly 
detects  (or  correctly  not  detects) . 
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g.  Total  Information  Quality  For  Each  Sensor 

Next,  we  calculate  the  total  value  for  each 
sensor  by  simply  finding  the  sum  of  all  the  initial 

information  component  values: 

total  value  sensory  =  location  accuracy  y  +  timeliness y  +  completeness y 

These  total  values  for  each  sensor  combine  with 
respect  to  the  sensor  detection  outcomes  to  become  the 
expected  return  values.  For  example,  with  two  sensors,  the 
detection  outcome  of  both  sensors  correctly  detecting  the 
target  has  an  expected  return  value  equal  to: 

total  value  sensory  +  total  value  sensor2  .  For  the  same  example,  the 

detection  outcome  of  sensori  correctly  detecting  the  target 
and  sensor2  incorrectly  not  detecting  the  target  has  an 
expected  return  value  equal  to: 

total  value  sensory  +  {total  value  sensor^  =01)  .  Therefore,  the  outcome 

when  both  sensors  agree  and  correctly  detect  the  target 
receives  the  larger  expected  return  value.  The  expected 
return  values  become  an  input  for  the  decision  tree  in 

Model  Component  III. 


E.  MODEL  COMPONENT  III,  DECISIONS  AND  OUTPUT 

I.  II. 
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The  third  component  of  the  model  integrates  the 
operational  decision  (from  a  decision  tree) ,  sensors' 
reports,  and  the  actual  target  state.  The  output  is  the 
MOE  for  this  thesis:  the  proportion  of  time  the  sensors' 

reports  agree  with  the  commander' s  decision,  and  they  are 
both  correct  (  'correct'  meaning  they  agree  with  the  actual 
target  state) .  Further  explanation  of  the  output  and  MOE 
occurs  later  in  this  section. 

1 .  Fuse  Information 

A  decision  tree  performs  data  fusion  of  multiple 
sensors'  detection  capabilities  and  information  quality 
values  in  order  to  represent  a  commander' s  decision  given  a 
common  operational  picture.  The  data  fusion  process  yields 
a  common  operational  picture  that  a  commander  uses  to  make 
an  operational  decision  to  attack  or  not.  This  model 
represents  data  fusion  and  decision  making  with  a  decision 
tree  (Marshall  and  Oliver,  1995) . 
a.  Decision  Tree 

The  decision  tree  requires  two  sets  of  inputs: 
detection  probabilities  (from  Model  Component  I)  and 
expected  return  values  (from  Model  Component  II) .  The 
following  decision  tree  illustrates  how  the  detection 
probabilities  (e.g.  Pr(Sl  detects),  Pr(S2  detects),  etc.) 
and  expected  return  values  (e.g.  A,  B,  etc.)  result  in  a 
decision  to  attack  or  not. 
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(SI  detects,  S2  detects,  S3  detects)  =  A 

(SI  detects,  S2  detects,  S3  not  detect)  =  B 
(SI  detects,  S2  not  detect,  S3  detects)  =  C 

(SI  detects,  S2  not  detect,  S3  not  detect)  =  D 

(SI  not  detect,  S2  detects,  S3  detects)  ^  E 

(SI  not  detect,  S2  detects,  S3  not  detect)  F 
(SI  not  detect,  S2  not  detect,  S3  detects)  =  G 

(SI  not  detect,  S2  not  detect,  S3  not  detect)  =  H 

(SI  detects,  S2  detects,  S3  detects)  =  I 

(SI  detects,  S2  detects,  S3  not  detect)  =  J 
(SI  detects,  S2  not  detect,  S3  detects)  =  K 

(SI  detects,  S2  not  detect,  S3  not  detect)  =  L 

(SI  not  detect,  S2  detects,  S3  detects)  =  M 

(SI  not  detect,  S2  detects,  S3  not  detect)  =  N 
(SI  not  detect,  S2  not  detect,  S3  detects)  =  O 

(SI  not  detect,  S2  not  detect,  S3  not  detect)  =  P 


Figure  1 0 . 


Decision  Tree  (After  Marshall  and  Oliver,  1995) 
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First  we  calculate  intermediate  expected  return 
values  AB  and  CD: 

AB  =  e\R{D  =  attack,  detects,  S2  detects)]  =  A*  Pr(S'3  detects)  +  5  *  Pr(5'3  not  detect) 

CD  =  e\R{D  =  attack,  5*1  detects,  SI  not  detect)]  =  C  *  Pr(S'3  detects)  +  Z)  *  Pr(5'3  not  detect) 

Next  we  calculate  intermediate  expected  return 
values  EF  and  GH : 

EE  =  e\R{D  =  attack,  5*1  not  detect,  S2  detects)]  =  E*  Pr(S'3  detects)  +  F  *  Pr(5'3  not  detect) 

GH  =  e\R{D  =  attack,  5*1  not  detect,  S2  not  detect)]  =  G  *  Pr(S'3  detects)  +  //  *  Pr(5'3  not  detect) 

Next  we  calculate  subsequent  expected  return 

values  ABCD  and  EFGH: 

ABCD  =  e\R{D  =  attack,  detects)]  =  AB  *  Pr(5'2  detects)  +  CD  *  Pr(5'2  not  detect) 

EFGH  =  E\R{D  =  attack,  not  detect)]  =  EF  *  Pr(5'2  detects)  +  GH  *  Pr(5'2  not  detect) 

Einally  we  calculate  the  final  expected  return 
value  ATTACK: 

ATTACK  =  e\R{D  =  attack)\  =  ABCD  *  Pr(5'l  detects)  +  EFGH  *  Pr(5'l  not  detect) 

Similar  calculations  yield  final  expected  return 
value  NO  ATTACK: 

NO  ATTACK  =  E[R{D  =  no  attack)]  =  IJKL  *  Vv(S\  detects)  +  MNOP  *  Vv(S\  not  detect) 

The  decision  tree  yields  the  expected  return 
values  for  the  decision  to  attack  or  not.  If  the  expected 
return  for  attacking  is  larger  than  the  expected  return  for 
not  attacking,  then  the  resulting  operational  decision  is 
to  attack.  Alternately,  if  the  expected  return  for 
attacking  is  less  than  or  equal  to  the  expected  return  for 
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not  attacking,  then  the  operational  decision  is  to  not 
attack . 

2 .  Decisions 

In  order  to  assess  the  operational  decision  (correct 
or  incorrect  based  on  target  presence) ,  we  compare  the 
decision  with  the  actual  state  of  the  target  (whether  or 
not  a  target  is  present) . 

a.  Actual  Target  State 

We  determine  the  actual  state  of  the  target  as 
follows:  if  a  random  draw  of  a  uniform(a  =  0,b  =  1)  is  less  than 

or  equal  to  the  probability  a  target  is  present  (input  from 
Model  Component  I),  then  the  target  is  present.  Otherwise, 
the  target  is  not  present.  If  the  decision  is  to  attack, 
and  the  target  is  actually  present,  then  a  correct 
operational  decision  was  made. 

ib.  Sensor  "Decision"  of  the  Target  State 

To  assess  each  sensor's  performance,  we  first 
calculate  each  sensor' s  "decision"  about  a  target— whether 
the  sensor  says  a  target  is  present  or  not.  For  each 
sensor,  we  compare  a  random  draw  of  a  uniform(a  =  0,b  =  1)  with 
the  sensor's  detection  probability  (input  from  Model 
Component  I)  .  If  the  random  number  is  less  than  or  equal 
to  the  detection  probability,  then  the  sensor  "decides"  it 
detected  a  target.  Otherwise,  the  sensor  "decides"  it  did 
not  detect  a  target. 

c.  Commander's  Operational  Decision 

In  order  to  assess  the  operational  decision 
(correct  or  incorrect  based  on  target  presence) ,  we  compare 
the  decision  with  the  actual  state  of  the  target  (whether 
or  not  a  target  is  present)  .  For  example,  if  the  decision 
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is  to  attack,  and  the  target  is  actually  present,  then  the 
commander  made  a  correct  operational  decision. 

3.  Measure  of  Effectiveness 

a.  Actual  Target  State,  Sensor  "Decision" ,  and 
Commander's  Decision  All  Agree 

Our  focus  for  this  thesis  is  all  the  instances 
where  a  sensor  agrees  with  the  operational  decision.  This 
focus  assists  in  the  effort  to  evaluate  how  sensor 
performance  supports  the  operational  decision.  A  sensor 
and  the  operational  decision  agree  when  the  sensor 
"decides"  a  target  is  present  and  the  decision  is  to 
attack,  or  when  the  sensor  "decides"  a  target  is  not 
present  and  the  decision  is  to  not  attack. 

In  addition  to  the  instances  where  a  sensor 
agrees  with  the  operational  decision,  we  want  to  know  when 
they  agree  correctly  and  when  they  agree  incorrectly.  In 
other  words,  we  compare  the  actual  target  state  (target 

present  or  not)  with  the  sensor  "decision"  about  the 
target,  and  with  the  operational  decision  to  attack  or  not. 
This  leads  to  the  following  mutually  exclusive  outcomes  for 
any  iteration  of  the  model  algorithm: 

•  Number  Correct:  measures  the  proportion  of  time 

the  given  sensors  support  the  decision,  and  the 
sensors  and  decision  are  correct 

•  Number  Incorrect:  measures  the  proportion  of 

time  the  given  sensors  support  the  decision,  and 
the  sensors  and  decision  are  incorrect 

•  Number  Sensor  Incorrect,  but  Decision  Correct: 

measures  the  proportion  of  time  the  given  sensors 
are  all  incorrect,  but  the  decision  is  correct 

•  Number  Sensor  Correct,  but  Decision  Incorrect: 

measures  the  proportion  of  time  the  given  sensors 
are  all  correct,  but  the  decision  is  incorrect. 
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The  first  of  these.  Number  Correct ,  is  the  MOE 
for  this  thesis.  It  represents  sensor  performance  with 
respect  to  the  commander's  decision.  The  MOE,  Number 
Correct ,  combines  the  ideas  of  sensitivity  ("...the 
proportion  of  positive  results  that  agree  with  the  true 
state")  and  specificity  ("...the  proportion  of  negative 
results  that  agree  with  the  true  state")  to  measure  all 
results  that  agree  with  the  true  state  (Bishop,  Elenberg, 
and  Holland,  1975)  .  (The  term,  results,  stands  for 

instances  where  the  sensors  and  commander' s  decision 
agree.)  High  MOE  values  represent  good  performance  and  low 
values  represent  poor  performance. 

The  following  figures  represent  what  makes  up 

each  of  the  mutually  exclusive  outcomes  listed  above.  The 
four  figures  below  combine  to  form  all  the  mutually 
exclusive  and  exhaustive  outcomes  for  the  model  in  this 

study . 

The  figure  below  displays  all  possible  model 
results  where  the  operational  decision  and  at  least  one 
sensor' s  "decision"  is  correct  for  a  situation  with  three 
sensors.  Again,  Number  Correct  measures  the  proportion  of 
time  the  given  sensors  support  the  decision,  and  the 

sensors  and  decision  are  correct.  Number  Correct 

encompasses  all  the  possible  outcomes  shown  in  the 

following  figure. 


32 
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The  next  figure  displays  the  only  possible  model 
result  where  the  operational  decision  is  correct  and  all 
the  sensors'  "decisions"  are  incorrect  for  a  situation  with 
three  sensors.  Again,  the  Number  Sensor  Incorrect,  but 
Decision  Correct  outcome  measures  the  proportion  of  time 
the  given  sensors  are  all  incorrect,  but  the  decision  is 
correct . 
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Figure  13.  Sensors  Incorrect /Decision  Correct 

The  following  figure  displays  the  only  possible 
model  result  where  the  operational  decision  is  incorrect 
and  all  the  sensors'  "decisions"  are  correct  for  a 
situation  with  three  sensors.  Again,  the  Number  Sensor 
Correct,  but  Decision  Incorrect  outcome  measures  the 
proportion  of  time  the  given  sensors  are  all  correct,  but 
the  decision  is  incorrect. 
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Assume  a  military  commander  faces  a  decision:  to 
attack  an  entity  or  not.  The  commander  receives 
information  from  three  sensors:  sensor  A  says  the  entity 
is  an  enemy  tactical  ballistic  missile  launcher,  sensor  B 
calls  it  an  unknown  object  emanating  heat,  and  sensor  C 
says  it's  a  vehicle  of  some  kind  with  electronic  emissions. 
Which  sensor  should  the  commander  trust  more?  Should  the 
commander  trust  all  equally?  What  other  information 
(experience,  previous  reports,  target  priority  list) 
besides  these  sensor  reports  does  the  commander  have  to 
help  form  his  decision?  Is  the  commander  biased  toward  one 
type  of  sensor  based  on  previous  experience?  Is  the 
commander  (or  sensor  operator)  well  trained  or  experienced? 
How  correct  is  each  of  the  sensors'  report?  How  reliable 
are  the  sensors?  Are  the  sensors  acting  void  of 
information,  or  are  they  passing  their  information  among 
one  another?  Are  they  manned  or  unmanned  sensors,  or  a 
combination  of  these  items?  What  criteria  do  the  sensors 
use  to  classify  an  entity  as  an  enemy,  unknown,  or  friend? 

Assume  the  commander  decides  to  attack  the  entity  and 
the  attack  successfully  eliminates  an  enemy  missile 
launcher.  Which  sensor  receives  the  most  credit  for  the 
successful  attack?  Or,  do  any  sensors  receive  any  credit? 
How  does  the  commander  credit  the  sensors  in  his  mind, 
thereby  adjusting  his  experience  base  and  influencing 
future  decisions? 

In  this  thesis  we  address  a  handful  of  the  many 
complex  relationships  mentioned  above.  For  example,  we 
assume  sensors  do  not  share  information  with  other  sensors, 
assume  no  bias  toward  a  sensor  by  the  commander,  assume  no 
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learning  or  experience  by  the  commander,  and  assume  no 
other  information  available  to  aid  in  decision  making. 

After  determining  sensor  performance,  we  apportion 
credit  to  sensors  based  on  that  performance.  For  example, 
a  three-sensor  combination  of  type  1  sensor,  type  2  sensor, 
and  type  3  sensor  (annotated  as  '123'  in  the  following 
figure)  provides  a  certain  performance  (approximately  860 
in  the  figure  below) . 


Sensor  Performance 
(Each  sensor  w/  a  unique  type) 


o 

O 

c 

(Z 

E 

o 

o 

Q. 


Si 

c 

0) 


•  E(performance) 


Sensor  Type  Combinations 


Figure  15.  Sensor  Credit  Apportionment  Example 

Each  individual  sensor  receives  credit  based  on  its 
individual  performance.  In  the  example  above,  the  type  1 
and  type  2  sensors  receive  the  least  credit  for  their 
contribution  to  sensor  type  combination  '123'  performance, 
and  the  type  3  sensor  receives  the  most  credit.  In  other 
words,  the  sensors  receive  a  ranking  of  type  3,  type  2,  and 
type  1  in  order  from  best  to  worst  (with  types  2  and  1 
nearly  identical) .  However,  in  this  particular  example, 
each  of  the  sensors  do  not  vary  significantly  at  O’ =  0.05 ,  as 
indicated  by  the  95%  confidence  intervals  shown  above.  In 
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general,  sensors  with  larger  performance  means  and  smaller 
performance  standard  deviations  receive  more  credit. 

A  brief  discussion  is  warranted  about  what  sensor 
"type"  means.  In  this  thesis,  a  sensor's  specifications 
define  its  type.  Typical  general  sensor  types  include  a 
person,  unmanned  aerial  vehicle  (UAV) ,  or  satellite. 
However,  each  person  possesses  his  or  her  own  sensor 
specifications,  so  each  person  also  makes  up  a  separate 
type  of  sensor.  Similarly,  different  UAV  types  or 

different  satellite  types  all  possess  their  own  sensor 
specifications;  therefore,  each  different  UAV  or  satellite 
also  makes  up  a  separate  type  of  sensor.  Even  with 
identical  sensor  specifications  between  two  sensors,  their 
performance  varies  due  to  randomness. 
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Ill .  SIMULATION  RESULTS  AND  ANALYSIS 


A.  GENERAL  SIMULATION  INFORMATION 

The  simulation  in  this  thesis  uses  up  to  three  sensors 
of  up  to  three  possible  types.  Each  simulation  replication 
yields  a  single  binary  value  among  four  mutually  exclusive 
and  exhaustive  alternatives: 

•  The  actual  target  state,  operational  decision, 

and  sensors  "decisions"  agree  (operational 
decision  and  sensors  correct) . 

•  The  actual  target  state  disagrees  with  the 
operational  decision  and  sensors  "decisions" 
(operational  decision  and  sensors  incorrect)  . 

•  The  actual  target  state  and  operational  decision 

agree,  but  the  sensors  "decisions"  disagrees 
(operational  decision  correct,  but  sensors 
incorrect) . 

•  The  actual  target  state  and  sensors  "decisions" 

agree,  but  the  operational  decision  disagrees 
(operational  decision  incorrect,  but  sensors 
correct) . 

The  first  alternative  above  makes  up  the  simulation 

measure  of  effectiveness  (MOE)  and  the  experiment's 

response:  the  proportion  of  correct  sensor  and  commander 
decisions  to  the  total  number  of  replications.  This 
proportion  represents  sensor  performance.  As  shown  and 
explained  in  the  previous  chapter,  the  following  figure 
displays  all  possible  outcomes  for  the  MOE  (the  figure 
shows  a  three  sensor  example) . 
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16. 

Sensors/Decision  Correct 

Similar 

figures 

exist  ( 

figures 

12,  13, 

and  14  in  the 

previous  chapter)  for  the  other  three  mutually  exclusive 
alternatives  listed  above.  For  each  replication  of  each 
run  of  the  simulation,  exactly  one  of  the  outcomes  from  the 
four  outcome  figures  occurs.  Our  primary  interest  in  this 
study  lies  in  those  outcomes  shown  above.  The  MOE  consists 
of  only  the  total  number  of  outcomes  from  the  figure  above 
that  occur  during  each  simulation  run. 

One  run  of  the  simulation  consists  of  1000 
replications  at  identical  initial  settings  for  each 
replication.  Each  of  the  1000  replications  varies  only  by 
randomness.  Each  run  yields  a  count  out  of  1000,  or  a 
proportion,  for  each  of  the  mutually  exclusive  and 
exhaustive  alternatives  listed  above.  Therefore,  the  four 
alternatives'  counts  sum  to  one  for  each  run.  This 
simulation  used  1000  replications  in  order  to  ensure  a 
confidence  interval  width  less  than  0.1.  An  estimate  for 
the  number  of  replications  required  for  an  interval  width 
of  0.1  yielded  a  value  of  approximately  384  replications. 
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However,  since  generating  data  from  the  simulation  required 
little  computing  time,  we  chose  1000  replications  (Devore, 
2000)  . 

The  simulation  settings  encompass  all  user-supplied 
inputs  such  as  sensor  types,  sensor  specifications,  and 
probability  distribution  parameters.  The  sensor  types  vary 
within  each  experiment.  The  sensor  specifications  and 
probability  distribution  parameters  remain  constant  for  all 
experiments.  The  sensor  specifications  used  appear  in 
Appendix  A.  The  parameters  for  the  probability 
distributions  appear  in  Appendix  B. 

B.  EXPERIMENTAL  DESIGN 

All  the  experiments  consist  of  the  three  sensors  as 
variables.  Each  sensor  has  four  possible  types:  type  1, 
type  2,  type  3,  or  'no  sensor'.  The  experiment's  response 
is  sensor  performance  (defined  above) . 

Each  of  the  experiments  conducts  a  stand-alone  look  at 
a  specific  aspect  of  the  relationship  among  the  sensor 
types  and  their  performance.  The  purpose  of  conducting  the 
experiments  in  this  manner  is  to  allow  for  model  face 
validation  and  verification  rather  than  trying  to  determine 
which  input  variables  are  most  significant  in  the  model. 
Therefore,  in  order  to  conduct  face  validation  and 
verification,  this  thesis  assumes  the  input  values  in 
Appendix  A  represent  realistic  values. 

All  the  experiments  consist  of  seven  runs,  nine  runs, 
or  19  runs,  depending  on  the  experiment  type  (discussed 
later  in  this  chapter)  .  The  seven-run  experiments  use 
seven  different  sensor  type  combinations— one  run  for  each 
combination,  while  the  19-run  experiment  uses  19  different 
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sensor  type  combinations.  Reference  Appendix  C  to  view 
each  experiment's  design. 

In  all  the  experiments  we  assume  all  sensors  involved 
are  within  detection  range  of  a  target  if  it  is  present. 
Also,  we  assume  no  multiple  targets  occur,  and  we  assume  a 
sensor  only  detects  a  single  target  once. 

Each  experiment's  response  represents  sensor 
performance.  High  response  represents  good  performance, 
and  low  response  represents  poor  performance.  The  response 
(or  sensor  performance)  ,  measures  the  proportion  of  correct 
sensor  and  commander  decisions  to  the  total  number  of 
events  (an  event  is  one  replication) .  For  example,  if  a 
sensor  and  the  commander  agree  a  target  is  present,  and  if 
the  target  really  is  present,  then  they  are  correct. 
Conversely,  if  they  agree  a  target  is  not  present,  and  if 
the  target  really  is  not  present,  then  they  are  also 
correct.  If  the  number  correct  within  a  run  is  350,  then 

350 

the  sensor  performance  for  that  run  is  - =  0.350.  The 

1000 

average  value  of  that  run  is  then  n  */>  =  1000  *  0.350  =  350  . 

C.  EXPERIMENT  A  (ALL  SENSORS  OF  SAME  TYPE) 

This  experiment's  purpose  is  to  show  that  sensors  of 
identical  type  provide  equivalent,  but  not  identical 
performance,  due  to  inherent  randomness.  This 

characteristic  models  reality:  two  identical  sensors 

perform  similarly,  but  not  identically,  in  separate  tests 
under  similar  circumstances.  Therefore,  results  from  this 
experiment  support  face  validation  of  this  model. 
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Figure  17.  Sensor  Performance  (All  Sensors  of  Type  1) 


Figure  18.  All  Sensors  of  Type  1  (Zoom  View) 
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Table  1.  Experiment  A  Summary  Statistics 


This  experiment  consists  of  a  mix  of  three  sensors  of 
identical  type  as  defined  by  each  sensor's  specifications. 
Appendix  A  contains  the  specifications  for  all  sensor  types 
used  for  this  thesis. 

The  figures  and  table  above  show  that  different 
sensors  of  identical  type  yield  similar  performance  over 
1000  replications.  In  other  words,  runs  1,  2,  and  3 
illustrate  that  three  different  sensors  of  identical  type 
produce  response  means  that  are  not  significantly  different 
at  « =  0.05  .  Again,  the  response  for  all  experiments 
represents  sensor  performance— large  response  values  are 
better.  Simulation  run  1  consists  of  only  sensor  1  set  at 
type  1,  run  2  consists  of  only  sensor  2  also  set  at  type  1, 
and  run  3  consists  of  only  sensor  3  set  at  type  1.  Their 
performance  after  1000  replications  measured  0.468,  0.452, 
and  0.466,  respectively.  These  results  support  the  idea 
that  different  sensors  of  identical  type  produce  similar 
performance  over  the  long  run.  The  differences  in  response 
means  and  confidence  intervals  occur  due  to  randomness  in 
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the  model.  Again,  each  replication  begins  with  identical 
initial  settings  and  varies  only  due  to  the  randomness  we 
input  (Appendices  A  and  B)  .  The  results  from  one 
replication  do  not  carry  forward  to  the  next  replication. 
The  results  from  this  experiment  support  common  sense:  if 
one  sensor  is  identical  to  another,  then  both  sensors' 
performance  should  agree  over  many  replications. 

The  figures  and  table  also  support  the  idea  that  three 
sensors  providing  information  to  a  commander  lead  to  better 
performance  over  the  long  run  than  any  of  the  individual 
sensors  working  alone.  Indeed,  this  is  the  premise  of  data 
fusion.  For  example,  the  response  mean  of  the  three 
sensors  combination  is  0.818,  while  the  response  means  of 
each  individual  sensor  are  0.468,  0.452,  and  0.466, 
respectively.  In  addition,  the  95%  confidence  interval  of 
three  sensor  combination  shows  over  the  long  run,  the  worst 
(lowest)  response  is  better  than  the  best  of  any  of  the 
individual  sensors  alone.  In  other  words,  at  a  =  0.05  their 
performance  differs  significantly. 

D.  EXPERIMENT  B  (EACH  SENSOR  OF  UNIQUE  TYPE) 

This  experiment's  purpose  is  to  compare  the 
performance  of  each  type  of  sensor  by  constraining  each 
sensor  to  only  one  type.  This  experiment  consists  of  three 
sensors  of  unique  types.  Each  sensor  maintains  its  type 
throughout  the  experiment.  For  instance,  sensor  2  is 
always  type  2  for  this  experiment. 
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Figure  19. 


Sensor  Performance  (1  Unique  Type  per  Sensor) 
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Figure  20 


1  Unique  Type  per  Sensor  (Zoom  View) 
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Table  2.  Experiment  B  Summary  Statistics 


In  this  experiment,  run  1  and  run  2  show  that  a  type  1 
sensor  performs  very  similarly  (response  means  not 
significantly  different  at  O’ =  0.05 )  to  a  type  2  sensor  even 
though  their  specifications  differ:  type  1  and  2  means  of 
0.468  and  0.469  and  95%  confidence  intervals  of  [0.437, 
0.499]  and  [0.438,  0.500],  respectively.  A  type  3  sensor 
appears  to  perform  better  than  either  of  the  other  two: 
type  3  mean  of  0.511  and  95%  confidence  interval  of  [0.480, 
0.542]  .  However,  at  « =  0.05 ,  its  mean  does  not  differ 
significantly  from  sensor  types  1  and  2. 

The  table  and  figures  above  continue  to  support  the 
idea  that  multiple  sensors  providing  information  to  a 
commander  lead  to  better  performance  over  the  long  run  than 
any  of  the  individual  sensors  working  alone.  Run  7 
illustrates  that  sensor  combination  of  type  1,  2,  and  3 
outperforms  any  of  the  other  combinations  with  a  mean  of 
0.860  and  95%  confidence  interval  of  [0.837,  0.880]. 
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E.  EXPERIMENT  C  (SENSOR  COMBINATION  ORDER) 

This  experiment  addresses  the  concern  of  whether  the 
order  of  sensor  settings  significantly  affects  the  results. 
By  'order'  we  mean  that  a  sensor  type  combination  of  type 
1,  2,  and  3,  has  a  different  order  than  a  combination  of 
type  3,  1,  and  2.  This  experiment  serves  two  purposes. 

First,  the  experiment  confirms  the  correctness  of  the 
model's  underlying  calculations.  Since  sensors  1,  2,  and  3 
are  all  type  1  sensors  during  runs  1,  2,  and  3,  their 
individual  performances  should  be  similar  but  not 
identical,  as  randomness  prevents  identical  performance 
from  identical  sensors.  In  other  words,  their  individual 
performances  are  equivalent. 

Second,  the  experiment  eliminates  the  need  for 
unnecessary  experiment  runs.  Since  sensors  of  the  same 
type  yield  equivalent  results,  a  sensor  type  combination  of 
type  1,  2,  and  3,  for  example,  yields  a  result  equivalent 
to  the  result  of  a  type  3,  1,  and  2  combination. 
Therefore,  the  'order'  of  the  sensor  types  does  not 
significantly  affect  the  results  of  the  experiment. 
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Figure  21.  Sensor  Performance  (Check  Combination  Order) 


Figure  22.  Check  Combination  Order  (Zoom  View) 
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Table  3.  Experiment  C  Summary  Statistics 


The  figures  above  illustrate  the  results  from  the 
table  that  sensors  of  the  same  type  yield  response  means 
that  are  not  significantly  different  at  « =  0.05  .  In  other 
words,  the  results  are  equivalent  for  sensors  of  the  same 
type . 

F.  EXPERIMENT  D  (IMPROVED  SENSOR) 

Intuition  indicates  an  improved  sensor  performs  better 
than  the  existing  sensors.  This  experiment  confirms  that 
intuition,  and  therefore  contributes  to  verifying  the 
underlying  model. 

By  improving  the  input  sensor  specifications  of  sensor 
1  (Appendix  A  displays  the  sensor' s  improved-type 
specifications)  ,  and  by  not  changing  the  other  sensors' 
specifications,  the  results  below  verify  expectation.  They 
support  intuition:  sensor  1  and  sensor  combinations 
involving  sensor  1  outperform  the  other  similar  sensor 
combinations  as  shown  in  the  figures  below.  Also, 
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comparison  of  the  figures  below  with  figures  19  and  20 
above  illustrate  sensor  I's  performance  changes  due  to 
improved  specifications.  That  supports  intuition  that 
improved  specifications  lead  to  improved  performance. 


Figure  23.  Sensor  Performance  (Sensor  1  Improved) 


Figure  24 . 


Sensor  1  Improved  (Zoom  View) 


G.  EXPERIMENT  E  (EACH  SENSOR  TYPE  VARIES) 

The  purpose  of  this  experiment  is  to  demonstrate  how 
the  sensor  credit  apportionment  method  from  this  thesis 
compares  performance  of  multiple  sensors  (three  sensors  in 
this  study)  of  multiple  types  (three  possible  types  in  this 
study)  .  The  experiment  yields  the  results  of  all 

combinations  of  three  sensors  of  three  possible  types. 

This  experiment  represents  how  this  credit 
apportionment  method  assists  decision  makers  visualize 
possible  performance  provided  by  different  sensor  type 
combinations  for  a  particular  set  of  input  values.  Input 
values  needed: 

•  Sensor  specifications  (Appendix  A) 

•  Probability  a  target  is  present  (Appendix  B) 

•  Probability  a  target  attribute  is  present  given  a 

target  is  present  (Appendix  B) 

•  Sensors'  miss  distance  data  (Appendix  B) 

•  Time  required  for  sensor  information  to  reach  the 

commander  (Appendix  B) 

For  this  thesis  we  estimated  all  the  input  values 
above;  however,  historical  research  or  field  tests  can 
yield  realistic  input  values  for  use  in  future  credit 
apportionment  experiments. 

This  experiment  consists  of  three  sensors,  each  of 
type  1,  2,  or  3 .  Since  the  sensor  combination  order  does 

not  affect  the  response  (as  explained  previously) ,  the 
following  combinations  exhaust  all  possible  groups  of 
sensor  types. 
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Figure  25.  Sensor  Performance  (Any  Sensor  Type) 


Figure  26. 


Any  Sensor  Type  (Zoom  View) 


jlation 

Run 

Sensor 

Combination 

Sensor 

Type 

Response 

Mean 

95%  Cl 

Lower 

95%  Cl 

Upper 

1 

Sensor 

1 

1 

0.468 

0.437 

0 .499 

2 

Sensors 

1 

& 

2 

1 

& 

1 

0 . 734 

0 . 706 

0 .760 

3 

Sensors 

1 

& 

2 

& 

3 

1 

& 

1 

& 

1 

0 . 817 

0 .792 

0 . 840 

4 

Sensors 

1 

& 

2 

& 

3 

1 

& 

1 

& 

2 

0.844 

0 . 820 

0 .865 

5 

Sensors 

1 

& 

2 

& 

3 

1 

& 

1 

& 

3 

0 . 851 

0 . 828 

0 . 872 

6 

Sensors 

1 

& 

2 

& 

3 

1 

& 

2 

& 

2 

0 . 851 

0 . 828 

0 . 872 

7 

Sensors 

1 

& 

2 

& 

3 

1 

& 

2 

& 

3 

0 .866 

0 . 843 

0 . 886 

8 

Sensors 

1 

& 

2 

& 

3 

1 

& 

3 

& 

3 

0 . 870 

0 . 848 

0 . 889 

9 

Sensor 

1 

2 

0.537 

0.506 

0 . 568 

10 

Sensors 

1 

& 

2 

2 

& 

1 

0 . 727 

0 . 699 

0 . 754 

11 

Sensors 

1 

& 

2 

2 

& 

2 

0 .765 

0 . 738 

0 .790 

12 

Sensors 

1 

& 

2 

2 

& 

3 

0 . 772 

0 . 745 

0.797 

13 

Sensors 

1 

& 

2 

& 

3 

2 

& 

2 

& 

2 

0.847 

0 . 823 

0 .868 

14 

Sensors 

1 

& 

2 

& 

3 

2 

& 

2 

& 

3 

0 . 877 

0 . 855 

0 .896 

15 

Sensors 

1 

& 

2 

& 

3 

2 

& 

3 

& 

3 

0 . 881 

0 . 859 

0 . 900 

16 

Sensor 

1 

3 

0.526 

0.495 

0 . 557 

17 

Sensors 

1 

& 

2 

3 

& 

1 

0 . 735 

0 . 707 

0 .761 

18 

Sensors 

1 

& 

2 

3 

& 

3 

0 .763 

0 . 736 

0 . 788 

19 

Sensors 

1 

& 

2 

& 

3 

3 

& 

3 

& 

3 

0 . 883 

0 .862 

0 . 901 

Table  4 


Experiment  E  Summary  Statistics 


Erom  the  figures  and  table  above,  all  combinations  of 
three  sensors  (except  for  the  combination  of  three  sensors 
of  type  1)  perform  best,  and  yield  response  means  that  are 
not  significantly  different  at  « =  0.05  .  Also,  all 

combinations  of  two  sensors  yield  response  means  that  are 
not  significantly  different  at  « =  0.05 ,  and  these  perform 


better  than  each  individual  sensor.  These  results  support 
the  idea  that  multiple  sensor  combinations  provide  better 
performance  (or,  better  assist  a  commander  with  decision 
making)  than  any  single  member  of  the  multiple  sensor 
combination  acting  alone.  In  other  words,  multiple  sensors 
provide  a  synergistic  effect  on  decision  making 
performance . 

The  individual  sensors  in  this  experiment  perform  very 
similarly.  At  « =  0.05  sensor  type  1  and  sensor  type  3 
provide  significantly  similar  response  means.  In  addition, 
sensor  type  2  and  sensor  type  3  do  not  differ  significantly 
at  « =  0.05  .  Therefore,  for  the  particular  set  of  input 
parameters  from  Appendices  A  and  B,  all  three  sensors 
provide  essentially  identical  performance  in  this 
experiment.  In  this  case  we  rank  all  three  sensors  the 
same  in  terms  of  apportioning  credit. 

H.  MODEL  VERIFICATION  AND  VALIDATION 

1 .  Verification 

The  Army  states  that  part  of  the  "...verification 
process...establishes  whether  the  M&S  [Model  &  Simulation] 
logic  and  code  correctly  perform  the  intended  functions" 
(Army  Regulation  5-11,  1997) .  Similarly,  Law  and  Kelton 
suggest  the  following  verification  technique:  "Run  the 
simulation  under  a  variety  of  settings  of  the  input 
parameters,  and  check  to  see  that  the  output  is  reasonable" 
(Law  and  Kelton,  2000) .  This  thesis  implements  that 
technique  by  varying  sensor  types  among  all  the 
experiments.  Although  all  the  above  experimental  results 
help  to  verify  the  underlying  logic  and  code.  Experiments  C 
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and  D  especially  indicate  appropriate  underlying 
calculations . 

2 .  Validation 

The  Army  defines  validation  as  "...the  process  of 
determining  the  extent  to  which  the  M&S  [Model  & 
Simulation]  accurately  represents  the  real  world  from  the 
perspective  of  its  intended  use"  (Army  Regulation  5-11, 
1997)  .  Assuming  the  input  values  for  the  model  represent 
realistic  values  reasonably  well,  the  model  results  agree 
with  what  we  would  expect  to  find.  However,  we  have 

examined  only  a  few  examples.  Therefore,  within  the  set  of 
examples  we  have  examined  in  this  thesis,  this  credit 
apportionment  model  adequately  "...represents  the  real  world 
from  the  perspective  of  its  intended  use"  (Army  Regulation 
5-11,  1997)  .  Or  stated  another  way,  each  of  the 

experiments  above  yield  results  "...consistent  with  perceived 
system  behavior..."  (Law  and  Kelton,  2000)  .  Law  and  Kelton 
define  this  as  face  validity. 

Although  true  validation,  which  requires  in  part 

"...high-quality  information  and  data  on  the  system,  "  is  not 
realistically  feasible  at  this  time,  further  research  for 
more  realistic  model  input  values  would  provide  a  more 
confident  face  validation  (Law  and  Kelton,  2000)  .  Law  and 
Kelton  also  state,  "The  most  definitive  test  of  a 
simulation  model's  validity  is  to  establish  that  its  output 
data  closely  resemble  the  output  data  that  would  be 
expected  from  the  actual...system"  (Law  and  Kelton,  2000)  . 
Since  an  actual  system  to  apportion  credit  to  multiple 
sensors  does  not  exist,  "...results  validation"  is  not 
feasible  at  this  time  (Law  and  Kelton,  2000) . 


56 


IV.  CONCLUSIONS 


A.  PURPOSE 


Recall  that  this  thesis  addresses  the  multiple  sensor 
credit  apportionment  phenomenon  by  conceptualizing  its 
process  (knowledge) ,  developing  a  working  model 
(algorithm) ,  and  considering  required  input  data  (data) . 

Since  very  little  published  information  exists  about 
multiple  sensor  credit  apportionment  and  its  inherent 
processes,  a  significant  portion  of  this  research  effort 
attempts  to  communicate  the  essence  of  the  apportioning 
credit  phenomenon  and  its  link  to  data  fusion  and 
information  quality. 


This  thesis  focuses  on  the  development  of  an 

appropriate  algorithm  that  fuses  detection  information  from 
independent  sources  while  considering  information  quality 
in  order  to  make  a  decision,  assess  the  decision,  and 
apportion  credit.  This  complex  and  detailed  algorithm 

forms  the  basis  for  a  computer  simulation.  Three  model 
development  steps  make  up  the  algorithm  development  process 
for  this  thesis: 

•  determine  sensor  detection  capabilities, 

•  determine  information  quality  each  sensor 

provides , 

•  fuse  detection  capabilities  and  information  value 
from  multiple  sensors. 


The  first  model  development  step  determines  each 
sensor' s  detection  capabilities  by  combining  the  presence 
of  a  target,  the  presences  of  target  attributes,  and  the 
sensor's  ability  to  detect  those  attributes.  This  thesis 
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accomplishes  this  step  by  using  the  law  of  total 
probability . 

The  second  model  development  step  determines  how  much 
value  the  sensors'  detection  information  holds  for  a 
commander  by  calculating  an  information  quality  value  for 
each  sensor.  Information  quality  contains  three  components 
in  this  thesis:  accuracy,  timeliness,  and  completeness 
(Perry,  2  0  0  0)  . 

The  third  model  development  step  fuses  the  results 
from  steps  one  and  two  in  order  to  provide  information  to  a 
commander  for  decision  making.  The  correctness  of  the 
commander' s  decision  based  on  the  actual  target  state  and 
sensor-provided  information  provides  the  measure  for 
comparing  performance  of  various  sensor  combinations. 

Consideration  of  required  input  data  discovers  that 
the  credit  apportionment  model  in  this  thesis  depends 
heavily  on  input  data.  For  this  thesis,  the  input  data  is 
estimated.  Therefore,  while  this  model  does  not  provide 
realistic  performance  of  each  sensor,  it  does  provide 
insight  about  the  relationships  between  sensors,  the 
operational  decision,  and  appropriate  credit  given  to  the 
sensors  based  on  their  performance. 

The  examination  of  this  method  includes  conducting 
five  experiments  of  various  settings  of  three  variables. 
The  variables  represent  the  sensors,  and  the  settings 
represent  the  types  of  sensors.  The  response  and  measure 
of  effectiveness  for  all  five  experiments  is  sensor 
performance.  We  define  sensor  performance  as  the 
proportion  of  correct  sensor  and  commander  decisions  to  the 
total  number  of  replications  defines  sensor  performance. 
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Each  experiment  provides  verification  of  the  underlying 
calculations,  and  provides  example  output  results  based  on 
input  parameters . 

B .  FUTURE  STUDIES 

A  more  comprehensive  representation  of  the  multiple 
sensor  credit  apportionment  method  from  this  thesis 
requires  future  research  in  several  areas. 

a.  More  Detailed  Experimental  Design 

The  research  effort  in  this  thesis  focuses  on 
developing  a  complex  model  of  a  complex  phenomenon.  The 
analysis  is  relatively  straightforward  in  that  five 
experiments  are  conducted  with  very  few  variables  in  order 
to  demonstrate  face-validation  of  the  modeling  process. 
Future  research  could  include  using  this  model  for  a  more 
comprehensive  experiment  with  numerous  variables  (the 
model's  input  parameters)  and  examine  which  are  most 
significant . 

b.  Weighting  Information  Quality  Components 

Varying  the  weighting  of  each  information  quality 
component  (accuracy,  timeliness,  and  completeness)  may 
reveal  how  each  affects  decision  correctness.  For  example, 
if  timeliness  receives  the  largest  weight  for  importance, 
then  accuracy  and  completeness  may  suffer.  Results  from 
different  weighting  may  lead  to  alternate  apportionment  of 
credit  than  when  we  hold  all  weights  equal. 

c.  Operational  Factors 

Army  professionals  describe  operations  in  terms  of 
Mission,  Enemy,  Terrain,  Troops,  and  Time  Available  (METT- 
T)  in  order  to  plan  and  execute  better  operations  (EM  101- 
5,  1997)  .  Incorporating  operations  data  into  the  model  by 
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means  of  an  operational  factor  or  a  set  of  factors  would 
represent  the  operational  environment  that  the  sensors 
operate  in.  METT-T  provides  a  good  breakdown  of  an 
operation;  therefore,  operational  factors  representing 
METT-T  may  yield  more  meaningful  results  for  Army  decision 
makers.  The  critical— and  difficult— port ion  of  this  idea 
consists  of  using  realistic  operations  data  for  the 
operational  factors. 

d .  Commander  Learning 

Incorporating  learning  (or  experience)  in  a  future 
version  of  the  model  could  provide  a  better  representation 
of  a  commander's  decision  making.  The  present  model 
assumes  a  commander  makes  an  operational  decision  based  on 
sensors'  current  reported  information,  and  not  on  sensors' 
previous  performance.  In  reality,  a  commander  makes  a 
decision  based  on  many  factors,  including  experience  with 
similar  sensors  and  their  previous  performance. 

e.  Generalize  Code 

The  model  and  simulation  for  this  thesis  are  tied  to 
three  or  fewer  sensors  with  three  or  fewer  types,  one 
target,  and  three  or  fewer  target  attributes.  Larger-scale 
sensor  performance  comparison  requires  the  ability  to 
accommodate  greater  quantities  of  sensors,  sensor  types, 
targets,  and  target  attributes. 

f.  Spreadsheet  Version 

Although  the  simulation  implementing  this  thesis' 
model  was  coded  in  Java,  a  spreadsheet  version  is  also 
possible.  A  spreadsheet  version  may  provide  an  easier 
environment  to  anyone  desiring  to  examine,  understand,  or 
adjust  the  model's  underlying  relationships  and 
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calculat ions .  However,  a  spreadsheet  version  is  not 
conducive  to  scaiing  up. 

The  Java  source  code  for  the  modei  and  simuiation  in 
this  thesis  is  avaiiabie  by  contacting  CPT  Mason  Crow, 
United  States  Miiitary  Academy,  Department  of  Mathematicai 
Sciences,  West  Point,  New  York. 
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APPENDIX  A.  SENSOR  SPECIFICATIONS 


A.  SENSOR  TYPE  1 

1.  Detects  attribute  1  best: 

•  probability  sensor  detects  attribute  1  given 

attribute  1  present  =  0.95 

•  probability  sensor  detects  attribute  1  given 

attribute  1  NOT  present  =  0.10  (false  detection) 

2.  Detects  attribute  2  poorly: 

•  probability  sensor  detects  attribute  2  given 

attribute  2  present  =  0.45 

•  probability  sensor  detects  attribute  2  given 

attribute  2  NOT  present  =  0.25  (false  detection) 

3.  Detects  attribute  3  satisfactorily: 

•  probability  sensor  detects  attribute  3  given 

attribute  3  present  =  0.75 

•  probability  sensor  detects  attribute  3  given 

attribute  3  NOT  present  =  0.20  (false  detection) 

4.  Detects  target  given  detection  of  attributes: 

•  probability  sensor  detects  target  given  sensor 

detects  attributes  1,  2,  and  3  =  0.95 

•  probability  sensor  detects  target  given  sensor 

detects  attributes  2  and  3  only  =  0.75 

•  probability  sensor  detects  target  given  sensor 

detects  attributes  1  and  3  only  =  0.90 

•  probability  sensor  detects  target  given  sensor 

detects  attributes  1  and  2  only  =  0.85 

•  probability  sensor  detects  target  given  sensor 

detects  attributes  3  only  =  0.70 

•  probability  sensor  detects  target  given  sensor 

detects  attributes  2  only  =  0.60 

•  probability  sensor  detects  target  given  sensor 

detects  attributes  1  only  =  0.80 

•  probability  sensor  detects  target  given  sensor 

detects  no  attributes  =  0.025 
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B.  SENSOR  TYPE  2 

1.  Detects  attribute  1  satisfactorily: 

•  probability  sensor  detects  attribute  1  given 

attribute  1  present  =  0.60 

•  probability  sensor  detects  attribute  1  given 

attribute  1  NOT  present  =  0.20  (false  detection) 

2.  Detects  attribute  2  best: 

•  probability  sensor  detects  attribute  2  given 

attribute  2  present  =  0.90 

•  probability  sensor  detects  attribute  2  given 

attribute  2  NOT  present  =  0.05  (false  detection) 

3.  Detects  attribute  3  poorly: 

•  probability  sensor  detects  attribute  3  given 

attribute  3  present  =  0.55 

•  probability  sensor  detects  attribute  3  given 

attribute  3  NOT  present  =  0.30  (false  detection) 

4.  Detects  target  given  detection  of  attributes: 

•  probability  sensor  detects  target  given  sensor 

detects  attributes  1,  2,  and  3  =  0.90 

•  probability  sensor  detects  target  given  sensor 

detects  attributes  2  and  3  only  =  0.80 

•  probability  sensor  detects  target  given  sensor 

detects  attributes  1  and  3  only  =  0.68 

•  probability  sensor  detects  target  given  sensor 

detects  attributes  1  and  2  only  =  0.87 

•  probability  sensor  detects  target  given  sensor 

detects  attributes  3  only  =  0.60 

•  probability  sensor  detects  target  given  sensor 

detects  attributes  2  only  =  0.75 

•  probability  sensor  detects  target  given  sensor 

detects  attributes  1  only  =  0.65 

•  probability  sensor  detects  target  given  sensor 

detects  no  attributes  =  0.025 


66 


C.  SENSOR  TYPE  3 

1.  Detects  attribute  1  poorly: 

•  probability  sensor  detects  attribute  1  given 

attribute  1  present  =  0.50 

•  probability  sensor  detects  attribute  1  given 

attribute  1  NOT  present  =  0.30  (false  detection) 

2.  Detects  attribute  2  satisfactorily: 

•  probability  sensor  detects  attribute  2  given 

attribute  2  present  =  0.65 

•  probability  sensor  detects  attribute  2  given 

attribute  2  NOT  present  =  0.15  (false  detection) 

3.  Detects  attribute  3  best: 

•  probability  sensor  detects  attribute  3  given 

attribute  3  present  =  0.97 

•  probability  sensor  detects  attribute  3  given 

attribute  3  NOT  present  =  0.10  (false  detection) 

4.  Detects  target  given  detection  of  attributes: 

•  probability  sensor  detects  target  given  sensor 

detects  attributes  1,  2,  and  3  =  0.93 

•  probability  sensor  detects  target  given  sensor 

detects  attributes  2  and  3  only  =  0.85 

•  probability  sensor  detects  target  given  sensor 

detects  attributes  1  and  3  only  =  0.80 

•  probability  sensor  detects  target  given  sensor 

detects  attributes  1  and  2  only  =  0.70 

•  probability  sensor  detects  target  given  sensor 

detects  attributes  3  only  =  0.77 

•  probability  sensor  detects  target  given  sensor 

detects  attributes  2  only  =  0.65 

•  probability  sensor  detects  target  given  sensor 

detects  attributes  1  only  =  0.6 

•  probability  sensor  detects  target  given  sensor 

detects  no  attributes  =  0.025 
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D.  SENSOR  IMPROVED  TYPE 

1.  Detects  attribute  1  perfectly: 

•  probability  sensor  detects  attribute  1  given 

attribute  1  present  =  1.0 

•  probability  sensor  detects  attribute  1  given 

attribute  1  NOT  present  =  0.0  (false  detection) 

2.  Detects  attribute  2  perfectly: 

•  probability  sensor  detects  attribute  2  given 

attribute  2  present  =  1.0 

•  probability  sensor  detects  attribute  2  given 

attribute  2  NOT  present  =  0.0  (false  detection) 

3.  Detects  attribute  3  perfectly: 

•  probability  sensor  detects  attribute  3  given 

attribute  3  present  =  1.0 

•  probability  sensor  detects  attribute  3  given 

attribute  3  NOT  present  =  0.0  (false  detection) 

4.  Detects  target  given  detection  of  attributes: 

•  probability  sensor  detects  target  given  sensor 

detects  attributes  1,  2,  and  3=1.0 

•  probability  sensor  detects  target  given  sensor 

detects  attributes  2  and  3  only  =  0.66 

•  probability  sensor  detects  target  given  sensor 

detects  attributes  1  and  3  only  =  0.66 

•  probability  sensor  detects  target  given  sensor 

detects  attributes  1  and  2  only  =  0.66 

•  probability  sensor  detects  target  given  sensor 

detects  attributes  3  only  =  0.33 

•  probability  sensor  detects  target  given  sensor 

detects  attributes  2  only  =  0.33 

•  probability  sensor  detects  target  given  sensor 

detects  attributes  1  only  =  0.33 

•  probability  sensor  detects  target  given  sensor 

detects  no  attributes  =  0.0 
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APPENDIX  B.  INPUT  DISTRIBUTION  PARAMETERS 


A.  TARGET  PRESENCE  PROBABILITY 

•  Probability  target  present  =  uniform(0.0,  0.75) 

B.  ATTRIBUTE  PRESENCE  PROBABILITY  GIVEN  TARGET  PRESENCE 

1 .  Attribute  1 : 

•  probability  attribute  1  present  given  target 

present  =  triangle ( 0 . 6,  1.0,  0.8) 

•  probability  attribute  1  present  given  target  NOT 

present  =  triangle ( 0 . 1 ,  0.7,  0.5) 

2 .  Attribute  2 : 

•  probability  attribute  2  present  given  target 

present  =  triangle ( 0 . 6,  1.0,  0.9) 

•  probability  attribute  2  present  given  target  NOT 

present  =  triangle ( 0 . 0 ,  0.4,  0.1) 

3 .  Attribute  3 : 

•  probability  attribute  3  present  given  target 

present  =  triangle ( 0 . 6,  1.0,  0.75) 

•  probability  attribute  3  present  given  target  NOT 

present  =  triangle ( 0 . 0 ,  0.1,  0.0) 

C .  LOCATION  ACCURACY 

•  Miss  distance  =  normal (0.0,  1.0) 

D .  TIMELINESS 

•  Time  duration  for  sensor  information  to  reach 
commander  =  exponential (rate  =  0.2) 
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EXPERIMENT  C  (SENSOR  COMBINATION  ORDER) 


EXPERIMENT  D  (IMPROVED  SENSOR) 


Simulation 

Sensor  1 

Sensor  2 

Sensor  3 

1 

Run 

Type 

Type 

Type 

1 

1 

NA 

NA 

2 

NA 

2 

NA 

3 

NA 

NA 

3 

4 

1 

2 

NA 

5 

1 

NA 

3 

6 

NA 

2 

3 

7 

1 

2 

3 

Table  8 . 
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